Generalized Reinforcement Learning for Manipulation Skills – Combining Low-dimensional Bayesian Optimization with High-dimensional Motion Optimization

This paper addresses the problem of how a robot can autonomously improve a manipulation skill in an efficient and secure manner. Instead of using the standard reinforcement learning formulation where all objectives are defined in a single reward function, we propose a generalized formulation that consists of three components: 1) A known analytic cost function; 2) A black-box reward function; 3) A black-box binary success constraint. While optimization of the analytic cost function is inherently high-dimensional, in typical robot manipulation problems we may assume that the black-box reward and constraint only depend on a lower dimensional projection of the policy. With our formulation we can exploit this structure and propose a sample-efficient learning framework that iteratively improves the skill with respect to the objective functions under the condition that the success constraint is fulfilled. The analytic cost function is optimized with motion optimization methods over the high dimensional policy where the lower dimensional parameters are fixed. The black-box reward is optimized with constraint Bayesian optimization over the lowerdimensional parameter. During both improvement steps the success constraint is used to keep the optimization in a secure region and to clearly distinguish between motions that lead to success or failure. The learning algorithm is evaluated on simulated benchmark problems and real-world tasks like opening a door with a PR2.